Scale in Distributed Systems

In recent years, scale has become a factor of increasing importance in the design of distributed systems. The scale of a system has three dimensions: numerical, geographical, and administrative. The numerical dimension consists of the number of users of the system, and the number of objects and services encompassed. The geographical dimension consists of the distance over which the system is scattered. The administrative dimension consists of the number of organizations that exert control over pieces of the system. The three dimensions of scale aaect distributed systems in many ways. Among the aaected components are naming, authentication, authorization, accounting, communication, the use of remote resources, and the mechanisms by which users view the system. Scale aaects reliability: as a system scales numerically, the likelihood that some host will be down increases; as it scales geographically, the likelihood that all hosts can communicate will decrease. Scale also aaects performance: its numerical component aaects the load on the servers and the amount of communication; its geographic component aaects communication latency. Administrative complexity is also aaected by scale: administration becomes more diicult as changes become more frequent and as they require the interaction of diierent administrative entities, possibly with connicting policies. Finally, scale aaects heterogeneity: as the size of a system grows it becomes less likely that all pieces will be identical. This paper looks at scale and how it aaects distributed systems. Approaches taken by existing systems are examined and their common aspects highlighted. The limits of scalability in these systems are discussed. A set of principles for scalable systems is presented along with a list of questions to be asked when considering how far a system scales.